Skip to content

Displays conversion confidence scores and grades #34

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

fabianofranz
Copy link
Contributor

@fabianofranz fabianofranz commented Jun 24, 2025

RHELAI-4250

Adds a new section to the Knowledge notebook to display confidence scores and grades to each document conversion.

@fabianofranz
Copy link
Contributor Author

@alimaredia @JustinXHale ptal

@fabianofranz
Copy link
Contributor Author

Screenshot From 2025-06-24 12-50-24

@anastasds
Copy link
Contributor

anastasds commented Jun 25, 2025

I wonder whether we should really consider all conversions with confidence less than the mean should be called "low confidence".

I have to think about whether this would really be rigorous, but we could construct a confidence interval based on individual confidence scores as a measure of how well we think it converts any given page on average, and then label conversions below the interval.

What is done here would be probably be a fine place to start, though - probably better to start out being underconfident than overconfident.

Copy link
Contributor

@alimaredia alimaredia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with @anastasds here on being a good starting point. I think that integration with the illuminator code to make that review all be in one place is another step I would want to see in the future.

@alimaredia alimaredia merged commit ee41913 into instructlab:main Jun 26, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants